14 February 2002. Thanks to S.
As noted by S, a site's file tree may be sometimes examined by calling up its robots.txt if the site uses that method to control file access. Access to robots.txt:
http://www.site.name/robots.txt
See other examples below: NSA, FBI, Army, Air Force, House of Representatives, Department of Justice, US Courts, Treasury Department
http://www.whitehouse.gov/robots.txt
# robots.txt for http://www.whitehouse.gov/ User-agent: * Disallow: /cgi-bin Disallow: /search Disallow: /query.html Disallow: /help Disallow: /afac/index.htm/text Disallow: /afac/text Disallow: /appointments/text Disallow: /cea/text Disallow: /ceq/text Disallow: /contact/text Disallow: /dpc/text Disallow: /email/text Disallow: /energy/text Disallow: /espanol/text Disallow: /firstlady/images/text Disallow: /firstlady/news-speeches/releases/print/text Disallow: /firstlady/news-speeches/releases/text Disallow: /firstlady/news-speeches/speeches/print/text Disallow: /firstlady/news-speeches/speeches/text Disallow: /firstlady/news-speeches/text Disallow: /firstlady/photoessay/text Disallow: /firstlady/text Disallow: /fsbr/text Disallow: /government/handbook/text Disallow: /government/images/text Disallow: /government/text Disallow: /greeting/text Disallow: /history/art/images/text Disallow: /history/art/text Disallow: /history/eeobtour/images/text Disallow: /history/eeobtour/text Disallow: /history/firstladies/text Disallow: /history/presidents/text Disallow: /history/text Disallow: /history/tours/print/text Disallow: /history/tours/text Disallow: /history/whtour/images/text Disallow: /history/whtour/text Disallow: /holiday/text Disallow: /homeland/text Disallow: /infocus/defense/text Disallow: /infocus/economy/text Disallow: /infocus/education/states/text Disallow: /infocus/education/text Disallow: /infocus/energy/text Disallow: /infocus/environment/text Disallow: /infocus/faith-based/text Disallow: /infocus/medicare/text Disallow: /infocus/social-security/text Disallow: /infocus/tax-relief/text Disallow: /infocus/text Disallow: /kids/abc/text Disallow: /kids/album/text Disallow: /kids/barney/text Disallow: /kids/connection/text Disallow: /kids/contact/text Disallow: /kids/dreamteam/text Disallow: /kids/firstlady/text Disallow: /kids/guide/print/text Disallow: /kids/guide/text Disallow: /kids/holiday/text Disallow: /kids/india/text Disallow: /kids/mrscheney/text Disallow: /kids/ofelia/text Disallow: /kids/president/text Disallow: /kids/quiz/text Disallow: /kids/spotty/text Disallow: /kids/teeball/text Disallow: /kids/teeball2/text Disallow: /kids/teeball3/text Disallow: /kids/text Disallow: /kids/timeline/text Disallow: /kids/tour/text Disallow: /kids/vicepresident/text Disallow: /library/omb/text Disallow: /mrscheney/news/text Disallow: /mrscheney/text Disallow: /national-anthem/text Disallow: /nec/text Disallow: /news/briefings/print/text Disallow: /news/briefings/text Disallow: /news/freedominitiative/text Disallow: /news/images/text Disallow: /news/nominations/text Disallow: /news/orders/text Disallow: /news/press/radio/text Disallow: /news/press/text Disallow: /news/print/releases/text Disallow: /news/print/text Disallow: /news/proclamations/text Disallow: /news/radio/print/text Disallow: /news/radio/text Disallow: /news/releases/2001/01/images/print/text Disallow: /news/releases/2001/01/images/text Disallow: /news/releases/2001/01/print/text Disallow: /news/releases/2001/01/text Disallow: /news/releases/2001/02/images/print/text Disallow: /news/releases/2001/02/images/text Disallow: /news/releases/2001/02/print/text Disallow: /news/releases/2001/02/text Disallow: /news/releases/2001/03/images/print/text Disallow: /news/releases/2001/03/images/text Disallow: /news/releases/2001/03/print/text Disallow: /news/releases/2001/03/text Disallow: /news/releases/2001/04/images/print/text Disallow: /news/releases/2001/04/images/text Disallow: /news/releases/2001/04/print/text Disallow: /news/releases/2001/04/text Disallow: /news/releases/2001/05/images/print/text Disallow: /news/releases/2001/05/images/text Disallow: /news/releases/2001/05/print/text Disallow: /news/releases/2001/05/text Disallow: /news/releases/2001/06/images/print/text Disallow: /news/releases/2001/06/images/text Disallow: /news/releases/2001/06/print/text Disallow: /news/releases/2001/06/text Disallow: /news/releases/2001/07/images/print/text Disallow: /news/releases/2001/07/images/text Disallow: /news/releases/2001/07/print/text Disallow: /news/releases/2001/07/text Disallow: /news/releases/2001/08/images/print/text Disallow: /news/releases/2001/08/images/text Disallow: /news/releases/2001/08/print/text Disallow: /news/releases/2001/08/text Disallow: /news/releases/2001/09/images/print/text Disallow: /news/releases/2001/09/images/text Disallow: /news/releases/2001/09/print/text Disallow: /news/releases/2001/09/text Disallow: /news/releases/2001/10/images/print/text Disallow: /news/releases/2001/10/images/text Disallow: /news/releases/2001/10/print/text Disallow: /news/releases/2001/10/text Disallow: /news/releases/2001/11/images/print/text Disallow: /news/releases/2001/11/images/text Disallow: /news/releases/2001/11/print/text Disallow: /news/releases/2001/11/text Disallow: /news/releases/2001/12/images/print/text Disallow: /news/releases/2001/12/images/text Disallow: /news/releases/2001/12/print/text Disallow: /news/releases/2001/12/text Disallow: /news/releases/2002/01/images/print/text Disallow: /news/releases/2002/01/images/text Disallow: /news/releases/2002/01/print/text Disallow: /news/releases/2002/01/text Disallow: /news/releases/print/text Disallow: /news/releases/text Disallow: /news/reports/text Disallow: /news/text Disallow: /news/usbudget/blueprint/text Disallow: /news/usbudget/states/print/text Disallow: /news/usbudget/states/text Disallow: /nsc/text Disallow: /oa/foia/text Disallow: /oa/jobs/text Disallow: /oa/oapo/text Disallow: /oa/text Disallow: /omb/budget/fy2002/text Disallow: /omb/budget/text Disallow: /omb/bulletins/text Disallow: /omb/circulars/a001/text Disallow: /omb/circulars/a016/text Disallow: /omb/circulars/a019/text Disallow: /omb/circulars/a021/text Disallow: /omb/circulars/a025/text Disallow: /omb/circulars/a034/text Disallow: /omb/circulars/a045/text Disallow: /omb/circulars/a050/text Disallow: /omb/circulars/a076/text Disallow: /omb/circulars/a087/text Disallow: /omb/circulars/a089/text Disallow: /omb/circulars/a094/text Disallow: /omb/circulars/a097/text Disallow: /omb/circulars/a102/text Disallow: /omb/circulars/a11/text Disallow: /omb/circulars/a110/text Disallow: /omb/circulars/a119/text Disallow: /omb/circulars/a122/text Disallow: /omb/circulars/a123/text Disallow: /omb/circulars/a126/text Disallow: /omb/circulars/a127/text Disallow: /omb/circulars/a129/text Disallow: /omb/circulars/a130/text Disallow: /omb/circulars/a131/text Disallow: /omb/circulars/a133/text Disallow: /omb/circulars/a133_compliance/00/text Disallow: /omb/circulars/a133_compliance/text Disallow: /omb/circulars/a134/text Disallow: /omb/circulars/a135/text Disallow: /omb/circulars/text Disallow: /omb/credit.bak/text Disallow: /omb/credit/text Disallow: /omb/fedreg/text Disallow: /omb/financial/text Disallow: /omb/foia/text Disallow: /omb/gils/text Disallow: /omb/grants/text Disallow: /omb/inforeg/text Disallow: /omb/legislative/7day/text Disallow: /omb/legislative/paygo/text Disallow: /omb/legislative/sap/105-1/text Disallow: /omb/legislative/sap/105-2/text Disallow: /omb/legislative/sap/106-1/text Disallow: /omb/legislative/sap/106-2/text Disallow: /omb/legislative/sap/107-1/appropriations/text Disallow: /omb/legislative/sap/107-1/number/text Disallow: /omb/legislative/sap/107-1/subcommittee/text Disallow: /omb/legislative/sap/107-1/text Disallow: /omb/legislative/sap/107-2/text Disallow: /omb/legislative/sap/1997/text Disallow: /omb/legislative/sap/1998/text Disallow: /omb/legislative/sap/1999/text Disallow: /omb/legislative/sap/2000/text Disallow: /omb/legislative/sap/text Disallow: /omb/legislative/testimony/text Disallow: /omb/legislative/text Disallow: /omb/memoranda/text Disallow: /omb/mgmt-gpra/text Disallow: /omb/organization/text Disallow: /omb/procurement/text Disallow: /omb/pubpress/text Disallow: /omb/recruitment/text Disallow: /omb/reports/text Disallow: /omb/text Disallow: /omb/whatsnew/text Disallow: /onap/text Disallow: /pfiab/text Disallow: /president/100days/text Disallow: /president/american-flag/text Disallow: /president/attack-response/text Disallow: /president/domestic-gallery/text Disallow: /president/gallery/photoessay/text Disallow: /president/gallery/text Disallow: /president/heartland-tour-gallery/text Disallow: /president/holiday/cards/text Disallow: /president/holiday/cheer/text Disallow: /president/holiday/deck-halls/text Disallow: /president/holiday/decorations/text Disallow: /president/holiday/hanukkah/text Disallow: /president/holiday/tree/text Disallow: /president/holiday/whtree/text Disallow: /president/images/text Disallow: /president/independence-day/text Disallow: /president/international-gallery/text Disallow: /president/intl-gallery2/text Disallow: /president/intl-gallery3/text Disallow: /president/intl-gallery4/text Disallow: /president/intl-gallery5/text Disallow: /president/intl-gallery6/text Disallow: /president/presidential-homes/text Disallow: /president/putin-visit/text Disallow: /president/statedinner-mexico-200109/text Disallow: /president/statedinnerprep-mexico-200109/text Disallow: /president/statevisitday2-mexico-200109/TEMP/text Disallow: /president/statevisitday2-mexico-200109/text Disallow: /president/tee-ball-01/text Disallow: /president/tee-ball-02/text Disallow: /president/tee-ball-03/text Disallow: /president/text Disallow: /president/world-leaders/text Disallow: /response/diplomatic/text Disallow: /response/military/text Disallow: /response/text Disallow: /text Disallow: /vicepresident/images/text Disallow: /vicepresident/news-speeches/speeches/print/text Disallow: /vicepresident/news-speeches/speeches/text Disallow: /vicepresident/news-speeches/text Disallow: /vicepresident/photoessay/text Disallow: /vicepresident/text Disallow: /whmo/text User-agent: whsearch Disallow: /cgi-bin Disallow: /search Disallow: /query.html Disallow: /help Disallow: /sitemap.html Disallow: /privacy.html Disallow: /accessibility.html
National Security Agency
User-agent: * Disallow: /images/ Disallow: /templates/ Disallow: *.gif Disallow: notice.html Disallow: statistics.html
Federal Bureau of Investigation
User-Agent: Disallow:
US Army
http://www.army.mil/robots.txt
User-agent: * Disallow: /cgi-bin/ Disallow: /reports/ Disallow: /summary/ Disallow: /old_design_ahp/ Disallow: /beta/ Disallow: /Documentation/ Disallow: /old_design_ahp/ Disallow: /images/ Disallow: /logs/ Disallow: /monthly/ Disallow: /photos/ Disallow: /vetinfo/ Disallow: /100days/
US Air Force
#robots.txt to skip over CGI directories User-agent: * Disallow: /cgi-bin/ Disallow: /passcgi/ Disallow: /tmp/
House of Representatives
http://www.house.gov/robots.txt
# # No robots allowed in the following directories ! # User-agent: * Disallow: /htbin Disallow: /docs/ARCHIVE Disallow: /docs/apps Disallow: /docs/moved_sites Disallow: /docs/temp Disallow: /docs/test Disallow: /docs/sites/bin Disallow: /docs/sites/etc Disallow: /docs/sites/dev Disallow: /docs/sites/usr Disallow: /docs/sites/other/webassistance
Department of Justice
http://www.usdoj.gov/robots.txt
User-agent: * Disallow: /Admin/ Disallow: /cgi-bin/ Disallow: /help/ Disallow: /img/ Disallow: /gif/ Disallow: /ins/ Disallow: /gopherdata/ Disallow: /ojp/ Disallow: /wusage/ Disallow: /archive/ Disallow: /opa/pr/support/ User-agent: Netscape-Compass-Robot/Archive Disallow:
US Courts
http://www.uscourts.gov/robots.txt
User-Agent: Disallow:
Treasury Department
http://www.treasury.gov/robots.txt
User-agent: * Disallow: /cgi-bin/ Disallow: /getstats/ Disallow: /home-temp/ Disallow: /logs/ Disallow: /new.junk/ Disallow: /public/ Disallow: /statbot/ Disallow: /templates/ Disallow: /webcache/ Disallow: /test/
Cryptome
http://cryptome.org/robots.txt
# go away User-agent: * Disallow: /